Connect to dynamic simulation API with full parameters #37

thangqp · 2022-11-29T17:42:18Z

Provide DynamicMappingClient to communicate to the dynamic-mapping-server in order to get model groovy and parameters par
Provide TimeSeriesClient to communicate to the timeseries-server in order to send timeseries/timeline
Provide an internal ParametersService in order to manipulate and copy reference parameters files from resources/parameters directory to a local directory, i.e. /tmp/dynamic_simulation_xxx
Provide NotificationService for notifying messages
Database, persist simulation result as : resultUuid | timeSeriesUuid | timeLineUuid | status IN result table
Writing tests for all services and controllers and a merge .par files utile, i.e. XmlMerge

- correct the copy files into test file system

…tificationService for message queue

…lace *.par files

…l_parameters

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationService.java

jonenst · 2023-01-13T08:55:14Z

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationWorkerService.java

+                            notificationService.emitResultDynamicSimulationMessage(sendMessage);
+                            LOGGER.info("Dynamic simulation complete (resultUuid='{}')", resultContext.getResultUuid());
+                        })
+                        .toFuture().get();


to investigate, at least put a big comment

At the moment, I put only more comments

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationWorkerService.java

jonenst · 2023-01-16T09:17:33Z

src/main/resources/db/changelog/changesets/changelog_2021-10-19T14:07:51Z.xml

@@ -5,7 +5,7 @@
            <column name="result_uuid" type="UUID">
                <constraints nullable="false" primaryKey="true" primaryKeyName="resultPK"/>
            </column>
-            <column name="result" type="BOOLEAN"/>
+            <column name="result" type="UUID"/>


why not a separate changeset ?

2 - Do not send timeseries result into rabitmq

jonenst · 2023-01-23T08:59:18Z

src/test/resources/com/powsybl/config/test/config.yml

+  debug: true
+
+dynawaltz-default-parameters:
+  parametersFile: /home/phamquy/Projects/dynawo/test/models.par


is this used ?

No, in fact there is a service, i.e. ParametersService, which configures parameters for Dynawaltz. This service does not use "dynawaltz-default-parameters".
I will remove all params in "dynawaltz-default-parameters"

jonenst · 2023-01-23T08:59:29Z

src/test/resources/com/powsybl/config/test/config.yml

@@ -0,0 +1,11 @@
+dynawaltz:
+  homeDir: /home/runner/work/dynamic-simulation-server/dynawo


is this used ?

Yes, homDir is mandatory and loaded by DynaWaltzConfig for the DynaWaltzProvider.

In prod env, config.yml is in the config directory ~/.itool

In the next step, we will test with container.. so this config can be removed?

Is it used by Github CI ?

yes! it is used when launching tests.

geofjamg · 2023-01-23T20:53:56Z

src/main/java/com/powsybl/dynamicsimulation/groovy/GroovyCurvesSupplier.java

+/**
+ * @author Mathieu Bague <[email protected]>
+ */
+public class GroovyCurvesSupplier implements CurvesSupplier {


To remove and use core one when switching to 5.1.0

Noted by add comment into code

geofjamg · 2023-01-23T20:54:04Z

src/main/java/com/powsybl/dynamicsimulation/groovy/GroovyEventModelsSupplier.java

+/**
+ * @author Marcos de Miguel <demiguelm at aia.es>
+ */
+public class GroovyEventModelsSupplier implements EventModelsSupplier {


To remove and use core one when switching to 5.1.0

Noted by add comment into code

geofjamg · 2023-01-23T20:55:52Z

src/main/java/org/gridsuite/ds/server/dto/DynamicSimulationStatus.java

    RUNNING,
-    COMPLETED
+    CONVERGED,


To check but I think SUCCEED and FAILED is more accurate than CONVERGED and DIVERGED for a dynamic simulation.

But we can see that later

In my opinion, CONVERGED and DIVERGED are two sub-states of SUCCEED. In contrast, FAILED is another state which relates to a technical issue.

Ok to discuss later

geofjamg · 2023-01-23T20:56:51Z

src/main/java/org/gridsuite/ds/server/dto/dynamicmapping/Script.java

+
+    private String name;
+
+    private String parentName;


What are the purposes of parentName and current?

This Dto is from the one in dynamic-mapping-server, so may be some fields are not really useful in dynamic-simulation-server.
'parentName' is the name of mapping
'name' is the name of script given by dynamic-mapping-server during the generation
In fact, these two names are not used for the computation but may be useful for the log, so I will keep these fields.

The field 'current', as I see in the implementation ScriptServiceImpl#isScriptCurrent(Script) in dynamic-mapping-server, is used to say that the script generated at some moment is out of date or not (in comparing to the modified date of the mapping). This field is not used in dynamic-simulation-server since the script is always generated on-demand.

I will remove this field in this Dto in dynamic-simulation-server

geofjamg · 2023-01-23T20:58:36Z

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationResultContext.java

@@ -45,25 +59,42 @@ private static String getNonNullHeader(MessageHeaders headers, String name) {

    public static DynamicSimulationResultContext fromMessage(Message<String> message) {
        Objects.requireNonNull(message);
+
+        String parametersPayload = message.getPayload();
+        ByteArrayInputStream bytesIS = new ByteArrayInputStream(parametersPayload.getBytes());


For simplicity, I will change the code by typing the payload as byte[] instead of currently as String to avoid a wasting step of conversion to String.

geofjamg · 2023-01-23T21:39:45Z

src/test/resources/data/xml/input/models.par

@@ -0,0 +1,281 @@
+<?xml version="1.0" encoding="UTF-8"?>


Is this file useful ? already in resources in src/test/resources/data/ieee14/_01/input/models.par ?

The same name but the content may be different. It is used for testing the xml merge tool. In my opinion, it is better to separate test data.. So I will keep this file..

Anyway, the merge tool will be removed when the multi input-files implementation at dynawo finished

geofjamg · 2023-01-23T21:40:25Z

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationWorkerService.java

-    private void sendResultMessage(Message<String> message) {
-        OUTPUT_MESSAGE_LOGGER.debug("Sending message : {}", message);
-        publishResult.send("publishResult-out-0", message);
+        List<TimeSeries> timeSeries = new ArrayList(result.getCurves().values());


new ArrayList<>(

geofjamg · 2023-01-23T21:40:42Z

src/test/java/org/gridsuite/ds/server/controller/DynamicSimulationControllerIEEE14Test.java

+
+        // prepare expected result to compare
+        DynamicSimulationResult expectedResult = DynamicSimulationResultDeserializer.read(getClass().getResourceAsStream(Paths.get(DATA_IEEE14_BASE_DIR, testBaseDir, OUTPUT, RESULT_JSON).toString()));
+        String jsonExpectedTimeSeries = TimeSeries.toJson(new ArrayList(expectedResult.getCurves().values()));


new ArrayList<>(

geofjamg · 2023-01-23T21:41:22Z

src/test/java/org/gridsuite/ds/server/controller/DynamicSimulationControllerIEEE14Test.java

+        String jsonResultTImeSeries = TimeSeries.toJson(TIME_SERIES_MOCK_BD.remove(timeSeriesUuid));
+
+        // compare result only timeseries
+        ObjectMapper mapper = new ObjectMapper();


Inject Spring one instead of creating a new one?

geofjamg · 2023-01-23T21:42:06Z

...est/java/org/gridsuite/ds/server/service/client/dynamicmapping/DynamicMappingClientTest.java

+                                    true,
+                                    parametersFile);
+
+                            ObjectWriter ow = new ObjectMapper().writer().withDefaultPrettyPrinter();


Inject Spring object mapper?

2 - Partial implementation 'stop' endpoint

jonenst · 2023-01-26T11:15:07Z

add a comment in the code for all the things that must be deleted in the near future (config.yml /home/runner, groovy stuff copypasted)

thangqp · 2023-01-26T21:25:15Z

add a comment in the code for all the things that must be deleted in the near future (config.yml /home/runner, groovy stuff copypasted)

Done!

geofjamg · 2023-01-27T15:30:10Z

src/main/java/org/gridsuite/ds/server/service/parameters/impl/ParametersServiceImpl.java

+        XmlMerge xmlMerge = new SimpleXmlMerge();
+        try {
+            Document mergedDoc = xmlMerge.merge(dynamicParamsIs, eventsParamsIs);
+            xmlMerge.export(mergedDoc, Files.newOutputStream(tmpDir.resolve(modelParFileName)));


Output stream should be closed

sonarqubecloud · 2023-01-30T10:01:53Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
12 Code Smells

86.3% Coverage
0.0% Duplication

thangqp added the WIP label Nov 29, 2022

thangqp added 19 commits December 9, 2022 15:27

Writing test with full parameters for service

5c34298

Config dynawaltz in test environment

6c867ae

Correct style

35b9c7a

writing test and evolve worker service

bde07d2

load dynamically *.par files into config directory

fcb3e52

more dynamically *.par files in filelist

8544af0

Provide TimeSeriesService client

4f77201

- send time series and time line to time-series-server

9a60a2b

- correct the copy files into test file system

Remove unused imports

84bee70

Test CI non-blocking thread

56ac185

Provide DynamicMappingService client, ParametersService client and No…

58d35d2

…tificationService for message queue

Correct tests

1c607ae

Correct TimeSeriesService

7ecc52b

ParameterService now take the current default config file system to p…

0bb81e5

…lace *.par files

Correct checkstyle

ea64d49

code clean on test

c462ada

Extract into interfaces

1ca560d

Restructure packages and add licences

33d6035

Merge branch 'main' into branching_to_dynamic_simulation_API_with_ful…

7f769f8

…l_parameters

thangqp force-pushed the branching_to_dynamic_simulation_API_with_full_parameters branch from c8db96d to 7f769f8 Compare December 9, 2022 14:29

thangqp added 9 commits December 9, 2022 16:58

Try a single unit test for detect block() problem

36567fe

Try to subscribe() instead of block()

29d805b

Check whether dynamo shell exists at the beginning of test

fc6b904

Check whether dynamo shell exists at the end of setup

a1cd783

Check whether dynamo directory and shell exists at the beginning of test

59cdac8

Unzip dynawo zip file into the parent of current directory

6acb303

Enhance code coverage test

6bfccce

Enhance code coverage

2adf501

Check with/clean : false when checkout source

e2b29c0

jonenst reviewed Jan 13, 2023

View reviewed changes

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationService.java Outdated Show resolved Hide resolved

jonenst reviewed Jan 13, 2023

View reviewed changes

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationWorkerService.java Outdated Show resolved Hide resolved

jonenst reviewed Jan 13, 2023

View reviewed changes

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationWorkerService.java Outdated Show resolved Hide resolved

jonenst reviewed Jan 13, 2023

View reviewed changes

src/main/java/org/gridsuite/ds/server/service/DynamicSimulationWorkerService.java Outdated Show resolved Hide resolved

thangqp added 2 commits January 13, 2023 18:22

Revisite code following the comment by reviewer

c06e198

Add more comments on using toFuture().get() instead of block()

89b02dc

jonenst reviewed Jan 16, 2023

View reviewed changes

thangqp added 5 commits January 16, 2023 17:10

1 - Change type for result column in the second changeset

99d33f0

2 - Do not send timeseries result into rabitmq

Correct checkstyles

b06b615

Notification for a receiver

d889815

Clean DynamicSimulationResultSerializer.java

7bc3083

Add TimeSeriesGroupInfos following to timeseries-server DTO

e4092da

jonenst reviewed Jan 23, 2023

View reviewed changes

Remove unnecessary parameters in config.yml and parameters.json

92caf98

geofjamg requested changes Jan 23, 2023

View reviewed changes

1 - Switch using toFuture().get() to block() in consumRun()

f9ab2c0

2 - Partial implementation 'stop' endpoint

Correct following to comments of reviewers

886ca6f

geofjamg reviewed Jan 27, 2023

View reviewed changes

thangqp added 2 commits January 29, 2023 15:59

Change return type of endpoint getStatus() to DynamicSimulationStatus

2de1581

Close opened output stream

3143c41

geofjamg approved these changes Jan 30, 2023

View reviewed changes

jonenst approved these changes Jan 30, 2023

View reviewed changes

geofjamg merged commit 6ddc839 into main Jan 30, 2023

geofjamg deleted the branching_to_dynamic_simulation_API_with_full_parameters branch January 30, 2023 13:29

Tristan-WorkGH mentioned this pull request Oct 9, 2023

Fix Liquibase diff when JPA entities not modified #66

Open

		@@ -0,0 +1,11 @@
		dynawaltz:
		homeDir: /home/runner/work/dynamic-simulation-server/dynawo

Connect to dynamic simulation API with full parameters #37

Connect to dynamic simulation API with full parameters #37

Uh oh!

Conversation

thangqp commented Nov 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thangqp Jan 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thangqp Jan 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geofjamg Jan 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

thangqp commented Nov 29, 2022 •

edited

Loading

thangqp Jan 26, 2023 •

edited

Loading

thangqp Jan 26, 2023 •

edited

Loading

geofjamg Jan 23, 2023 •

edited

Loading